Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
253 views
in Technique[技术] by (71.8m points)

c# - ElasticSearch's dynamic index mapping dropping logs due to poor field names

I have a semi-large project that has been using nlog, and throughout I re-used alot of field names for different datatypes. I started to send my logs (including all log properties/fields) to ElasticSearch, and now its starting to haunt me. I noticed if ElasticSearch is unable to convert a field to it's dataType it will just drop the log entirely. The dynamic index mapping is deciding the dataType depending on what it's seen first.

Is there anyway I can tell the index to use a default dataType like string?

PS; The Instance is cloud hosted, I access it through Kibana, and I have no Idea where to find a log that tells me if/when it drops logs for parsing errors.

Edit

Index Mapping

PUT /indexName
{
  "mappings": {
      "properties": {
        "@domain": {
          "type": "keyword"
        },
        "@logTarget": {
          "type": "keyword"
        },
        "@logger": {
          "type": "keyword"
        },
        "@memUsage": {
          "type": "long"
        },
        "@processID": {
          "type": "integer"
        },
        "@serviceGUID": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date"
        },
        "level": {
          "type": "keyword"
        },
        "message": {
          "type": "text"
        }
      }
  }
}

Unfortunately I don't know the api for pushing logs to elastic but here are some examples of what they might look like. Theses aren't the best examples but they show the overlapping/re-use of field names.

Example 1:

//NLog structured log
logger.LogInfo("That Lady has {count} cats", 5);
//JSON Object
{
    "@domain": "Service1",
    "@logTarget": "None",
    "@logger": "AppName.Program.Main",
    "@memUsage": 100,
    "@processID": 17000,
    "@serviceGUID": 0,
    
    "level": "INFO",
    "message": "That Lady has 5 cats",
    
    "count": 5,
}

Example 2:

//NLog structured log
int count = 3;
logger.LogInfo("Loaded {count}", count + " dogs");
//JSON Object
{
    "@domain": "Service1",
    "@logTarget": "None",
    "@logger": "AppName.Program.Main",
    "@memUsage": 100,
    "@processID": 17000,
    "@serviceGUID": 0,
    
    "level": "INFO",
    "message": "Loaded "5 dogs"",
    
    "count": 5,
}

Example 3:

//NLog structured log
object count = nil;
logger.LogInfo("Value is {count}", count);
//JSON Object
{
    "@domain": "Service1",
    "@logTarget": "None",
    "@logger": "AppName.Program.Main",
    "@memUsage": 100,
    "@processID": 17000,
    "@serviceGUID": 0,
    
    "level": "INFO",
    "message": "Loaded "5 dogs"",
    
    "count": 5,
}

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)
等待大神答复

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

2.1m questions

2.1m answers

60 comments

57.0k users

...