I have a semi-large project that has been using nlog, and throughout I re-used alot of field names for different datatypes. I started to send my logs (including all log properties/fields) to ElasticSearch, and now its starting to haunt me. I noticed if ElasticSearch is unable to convert a field to it's dataType it will just drop the log entirely. The dynamic index mapping is deciding the dataType depending on what it's seen first.
Is there anyway I can tell the index to use a default dataType like string?
PS; The Instance is cloud hosted, I access it through Kibana, and I have no Idea where to find a log that tells me if/when it drops logs for parsing errors.
Edit
Index Mapping
PUT /indexName
{
"mappings": {
"properties": {
"@domain": {
"type": "keyword"
},
"@logTarget": {
"type": "keyword"
},
"@logger": {
"type": "keyword"
},
"@memUsage": {
"type": "long"
},
"@processID": {
"type": "integer"
},
"@serviceGUID": {
"type": "keyword"
},
"@timestamp": {
"type": "date"
},
"level": {
"type": "keyword"
},
"message": {
"type": "text"
}
}
}
}
Unfortunately I don't know the api for pushing logs to elastic but here are some examples of what they might look like. Theses aren't the best examples but they show the overlapping/re-use of field names.
Example 1:
//NLog structured log
logger.LogInfo("That Lady has {count} cats", 5);
//JSON Object
{
"@domain": "Service1",
"@logTarget": "None",
"@logger": "AppName.Program.Main",
"@memUsage": 100,
"@processID": 17000,
"@serviceGUID": 0,
"level": "INFO",
"message": "That Lady has 5 cats",
"count": 5,
}
Example 2:
//NLog structured log
int count = 3;
logger.LogInfo("Loaded {count}", count + " dogs");
//JSON Object
{
"@domain": "Service1",
"@logTarget": "None",
"@logger": "AppName.Program.Main",
"@memUsage": 100,
"@processID": 17000,
"@serviceGUID": 0,
"level": "INFO",
"message": "Loaded "5 dogs"",
"count": 5,
}
Example 3:
//NLog structured log
object count = nil;
logger.LogInfo("Value is {count}", count);
//JSON Object
{
"@domain": "Service1",
"@logTarget": "None",
"@logger": "AppName.Program.Main",
"@memUsage": 100,
"@processID": 17000,
"@serviceGUID": 0,
"level": "INFO",
"message": "Loaded "5 dogs"",
"count": 5,
}