Skip to content

Code

  • Storing structured data without schema changes
    This is just a very short blog around how to store data in a SQL database without having to worry too much about continual schema changes. Most people will know about different flavours of storing data, flat files (csv, hdf, parquet, which is my favourite), SQL, or NoSQL datastores. I shall skip graph stores in… Read More »Storing structured data without schema changes
  • The FREEZE Data Pipeline
    For the Arctic Risk data repository I decided to use a standard architecture often referred to as a data lake. we ingest live, soft real time data, using a suite of ingestion scripts (Continual or Warm), and we do one-off imports (Batch or Cold) The tool suite is centred around available open source tools. While… Read More »The FREEZE Data Pipeline
  • Geopandas: Earth as a Sphere
    This one kept me busy for actually a few months. Map projections, the art of mapping the surface of a sphere onto a two dimensional canvas, is a science in its own rights. Dealing with data from the arctic will cause grief, as most maps we commonly use, limit the range of latitudes displayed to… Read More »Geopandas: Earth as a Sphere