TransWikia.com

How to correctly map huge json file to Java (pojo)?

Stack Overflow Asked by Orkhan Hasanli on December 3, 2021

Large json files are available (from 100 MB to 2 GB).

How I can create java (pojo) from json? I need to get the names of all fields. For example, one class can have more than 1000 fields.

Tried the following libraries & online services:

JsonSchema2Pojo works well, but has a limit of about 50,000 characters at a time. Due to the presence of limits, I cannot completely map the class. Need to partially copy the json each time and then remove the duplicate fields.

http://www.jsonschema2pojo.org/

JsonToJava
https://github.com/astav/JsonToJava

Also tried several other online services. They also have limits or do not work correctly.

Example of json file:

{"type":"FeatureCollection","features":[
{"type":"Feature","geometry":{"type":"Point","coordinates":[27.5618791,53.902334]},"properties":{"ele":"280","name":"Минск","place":"city","capital":"yes","name:ar":"مينسك","name:be":"Мінск","name:bg":"Минск","name:bo":"མིན་སིཀ།","name:cs":"Minsk","name:cu":"Мѣньскъ","name:cv":"Минск","name:de":"Minsk","name:el":"Μινσκ","name:en":"Minsk","name:eo":"Minsko","name:es":"Minsk","name:et":"Minsk","name:fa":"مینسک","name:fi":"Minsk","name:fr":"Minsk","name:ga":"Minsc","name:gl":"Minsk - Мінск","name:he":"מינסק","name:hi":"मिन्‍स्‍क","name:hr":"Minsk","name:hu":"Minszk","name:hy":"Մինսկ","name:ia":"Minsk","name:io":"Minsk","name:is":"Minsk","name:it":"Minsk","name:ja":"ミンスク","name:ka":"მინსკი","name:kk":"Минск","name:kn":"ಮಿನ್ಸ್ಕ್","name:ko":"민스크","name:ku":"Mînsk","name:kv":"Минск","name:ky":"Минск","name:la":"Minscum","name:lt":"Minskas","name:lv":"Minska","name:mk":"Минск","name:ml":"മിൻസ്ക്","name:mr":"मिन्‍स्‍क","name:nl":"Minsk","name:no":"Minsk","name:oc":"Minsk","name:os":"Минск","name:pl":"Mińsk","name:pt":"Minsk","name:ru":"Минск","name:sk":"Minsk","name:sl":"Minsk","name:sr":"Минск","name:sv":"Minsk","name:ta":"மின்ஸ்க்","name:tg":"Минск","name:th":"มินสก์","name:tt":"Минск","name:ug":"مىنىسكى","name:uk":"Мінськ","name:ur":"منسک","name:vi":"Minxcơ","name:vo":"Minsk","name:yi":"מינסק","name:zh":"明斯克","website":"https://minsk.gov.by/","int_name":"Minsk","name:ast":"Minsk","name:ckb":"مینسک","name:csb":"Mińsk","name:jbo":"misk","name:mhr":"Минск","name:myv":"Минск ош","name:nds":"Minsk","name:pnb":"منسک","name:rue":"Мінск","name:sah":"Минскай","name:szl":"Mińsk","name:udm":"Минск","name:wuu":"明斯克","nat_name":"Мінск","old_name":"Менск","wikidata":"Q2280","wikipedia":"ru:Минск","population":"1982444","admin_level":"2","alt_name:be":"Менск","alt_name:vi":"Minsk;Minxcơva","name:prefix":"город","old_name:be":"Менск","addr:country":"BY","name:bat-smg":"Minsks","name:sr-Latn":"Minsk","wikipedia:be":"Мінск","wikipedia:en":"Minsk","wikipedia:pl":"Mińsk","addr:postcode":"220000","is_in:country":"Belarus","name:be-tarask":"Менск","source:name:oc":"Lo Congrès","is_in:continent":"Europe","population:date":"2018-01-01","capital_ISO3166-1":"yes","source:population":"Белстат","is_in:country_code":"BY"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5861178,53.9510657],[27.5861196,53.9510294],[27.5862319,53.9510313],[27.5862302,53.9510676],[27.5861178,53.9510657]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5885075,53.9512721],[27.5885398,53.9511614],[27.5887363,53.9511813],[27.588704,53.9512919],[27.5885075,53.9512721]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5881428,53.9512622],[27.5881704,53.9511135],[27.588421,53.9511296],[27.5883935,53.9512783],[27.5881428,53.9512622]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5892179,53.9516538],[27.5893274,53.9516011],[27.5893778,53.9516374],[27.5894352,53.9516098],[27.5895358,53.9516823],[27.5893689,53.9517626],[27.5892179,53.9516538]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.588949,53.9517486],[27.5891273,53.9516621],[27.5892764,53.9517686],[27.5890981,53.9518551],[27.588949,53.9517486]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5867595,53.9500825],[27.5867648,53.9499951],[27.5869291,53.9499985],[27.5869238,53.9500859],[27.5867595,53.9500825]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5830382,53.9503042],[27.5830422,53.9502013],[27.5832783,53.9502045],[27.5832743,53.9503074],[27.5830382,53.9503042]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5820906,53.9497446],[27.5820973,53.9496721],[27.5822371,53.9496765],[27.5822305,53.949749],[27.5820906,53.9497446]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5477403,53.8940388],[27.5482144,53.8939744],[27.5482459,53.8940791],[27.5482929,53.8940736],[27.5483083,53.8941424],[27.5482593,53.8941471],[27.5482741,53.8942048],[27.5485718,53.8943067],[27.5487113,53.8943028],[27.5487556,53.8943755],[27.548895,53.8944261],[27.5487985,53.894513],[27.5488079,53.8945841],[27.5483385,53.8943889],[27.548317,53.8943202],[27.5481439,53.8942549],[27.5481045,53.8941843],[27.5479863,53.8942063],[27.5479953,53.8942214],[27.5478248,53.8942506],[27.5477403,53.8940388]]]]},"properties":{"name":"Инженерный корпус метрополитена","building":"yes"}}
]}

Thanks in advance!

One Answer

Thanks a lot for your advices. Implemented in the following way: I used JsonAnySetter annotation to set all fields instead creating pojo with all of the possible fields. Here is my Dto's

FeatureDto

@Getter
@ToString
public class FeatureDto {

  @JsonProperty("type")
  private String type;
  @JsonProperty("geometry")
  private GeometryDto geometry;
  @JsonProperty("properties")
  private PropertiesDto properties;

}

PropertiesDto

@Getter
@ToString
public class PropertiesDto {

  private List<FieldDto> fieldDtos = new ArrayList<>();

  @JsonAnySetter
  public void setProperties(String code, Object value)
  {

    if(value instanceof ArrayList) {
      ArrayList<Object> list = (ArrayList<Object>) value;
      StringBuffer stringBuffer = new StringBuffer();
      list.forEach(o -> stringBuffer.append(o).append(", "));
      fieldDtos.add(new FieldDto(code, stringBuffer.toString()));
      return;
    }

    fieldDtos.add(new FieldDto(code, value.toString()));
  }

}

FieldDto

@Data
@ToString
@JsonIgnoreProperties(ignoreUnknown = true)
@AllArgsConstructor
public class FieldDto {

  @JsonProperty("code")
  private String code;
  @JsonProperty("value")
  private String value;

}

Here is the json source:

{"type":"FeatureCollection","features":[
{"type":"Feature","geometry":{"type":"Point","coordinates":[27.5618791,53.902334]},"properties":{"ele":"280","name":"Минск","place":"city","capital":"yes","name:ar":"مينسك","name:be":"Мінск","name:bg":"Минск","name:bo":"མིན་སིཀ།","name:cs":"Minsk","name:cu":"Мѣньскъ","name:cv":"Минск","name:de":"Minsk","name:el":"Μινσκ","name:en":"Minsk","name:eo":"Minsko","name:es":"Minsk","name:et":"Minsk","name:fa":"مینسک","name:fi":"Minsk","name:fr":"Minsk","name:ga":"Minsc","name:gl":"Minsk - Мінск","name:he":"מינסק","name:hi":"मिन्‍स्‍क","name:hr":"Minsk","name:hu":"Minszk","name:hy":"Մինսկ","name:ia":"Minsk","name:io":"Minsk","name:is":"Minsk","name:it":"Minsk","name:ja":"ミンスク","name:ka":"მინსკი","name:kk":"Минск","name:kn":"ಮಿನ್ಸ್ಕ್","name:ko":"민스크","name:ku":"Mînsk","name:kv":"Минск","name:ky":"Минск","name:la":"Minscum","name:lt":"Minskas","name:lv":"Minska","name:mk":"Минск","name:ml":"മിൻസ്ക്","name:mr":"मिन्‍स्‍क","name:nl":"Minsk","name:no":"Minsk","name:oc":"Minsk","name:os":"Минск","name:pl":"Mińsk","name:pt":"Minsk","name:ru":"Минск","name:sk":"Minsk","name:sl":"Minsk","name:sr":"Минск","name:sv":"Minsk","name:ta":"மின்ஸ்க்","name:tg":"Минск","name:th":"มินสก์","name:tt":"Минск","name:ug":"مىنىسكى","name:uk":"Мінськ","name:ur":"منسک","name:vi":"Minxcơ","name:vo":"Minsk","name:yi":"מינסק","name:zh":"明斯克","website":"https://minsk.gov.by/","int_name":"Minsk","name:ast":"Minsk","name:ckb":"مینسک","name:csb":"Mińsk","name:jbo":"misk","name:mhr":"Минск","name:myv":"Минск ош","name:nds":"Minsk","name:pnb":"منسک","name:rue":"Мінск","name:sah":"Минскай","name:szl":"Mińsk","name:udm":"Минск","name:wuu":"明斯克","nat_name":"Мінск","old_name":"Менск","wikidata":"Q2280","wikipedia":"ru:Минск","population":"1982444","admin_level":"2","alt_name:be":"Менск","alt_name:vi":"Minsk;Minxcơva","name:prefix":"город","old_name:be":"Менск","addr:country":"BY","name:bat-smg":"Minsks","name:sr-Latn":"Minsk","wikipedia:be":"Мінск","wikipedia:en":"Minsk","wikipedia:pl":"Mińsk","addr:postcode":"220000","is_in:country":"Belarus","name:be-tarask":"Менск","source:name:oc":"Lo Congrès","is_in:continent":"Europe","population:date":"2018-01-01","capital_ISO3166-1":"yes","source:population":"Белстат","is_in:country_code":"BY"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5861178,53.9510657],[27.5861196,53.9510294],[27.5862319,53.9510313],[27.5862302,53.9510676],[27.5861178,53.9510657]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5885075,53.9512721],[27.5885398,53.9511614],[27.5887363,53.9511813],[27.588704,53.9512919],[27.5885075,53.9512721]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5881428,53.9512622],[27.5881704,53.9511135],[27.588421,53.9511296],[27.5883935,53.9512783],[27.5881428,53.9512622]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5892179,53.9516538],[27.5893274,53.9516011],[27.5893778,53.9516374],[27.5894352,53.9516098],[27.5895358,53.9516823],[27.5893689,53.9517626],[27.5892179,53.9516538]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.588949,53.9517486],[27.5891273,53.9516621],[27.5892764,53.9517686],[27.5890981,53.9518551],[27.588949,53.9517486]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5867595,53.9500825],[27.5867648,53.9499951],[27.5869291,53.9499985],[27.5869238,53.9500859],[27.5867595,53.9500825]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5830382,53.9503042],[27.5830422,53.9502013],[27.5832783,53.9502045],[27.5832743,53.9503074],[27.5830382,53.9503042]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5820906,53.9497446],[27.5820973,53.9496721],[27.5822371,53.9496765],[27.5822305,53.949749],[27.5820906,53.9497446]]]]},"properties":{"building":"yes"}},
{"type":"Feature","geometry":{"type":"MultiPolygon","coordinates":[[[[27.5477403,53.8940388],[27.5482144,53.8939744],[27.5482459,53.8940791],[27.5482929,53.8940736],[27.5483083,53.8941424],[27.5482593,53.8941471],[27.5482741,53.8942048],[27.5485718,53.8943067],[27.5487113,53.8943028],[27.5487556,53.8943755],[27.548895,53.8944261],[27.5487985,53.894513],[27.5488079,53.8945841],[27.5483385,53.8943889],[27.548317,53.8943202],[27.5481439,53.8942549],[27.5481045,53.8941843],[27.5479863,53.8942063],[27.5479953,53.8942214],[27.5478248,53.8942506],[27.5477403,53.8940388]]]]},"properties":{"name":"Инженерный корпус метрополитена","building":"yes"}}
]}

Answered by Orkhan Hasanli on December 3, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP