IEEE Computational Intelligence Magazine - November 2020 - 55

violence. This is an example of a complex
... the MIDAS platform could be utilized to enhance
and multi-level issue where a range of heterogeneous data is required at a policy-making
our understanding of the adverse collateral effects and
level to enable better analysis and visualizalasting impact of the COVID-19 pandemic lockdown.
tion of the situation. The hindering factor for
the real use of the MIDAS platform in mental health cases in Finland is the current legrelationship of clinical variables and open data indicators
islation often restricting the combined usage of the anonymized
(i.e. COVID-19-related indicators such as number of cases,
social and healthcare data collected on individuals in policy
intensive care units or beds, and recovered people that have
level decision making.
been published through different organizational open data
As an example of how the MIDAS platform could be used
agencies) at an aggregated location level (e.g. primary care
in the COVID-19 context, the lockdown effects on childunit or trust). The COVID-19 pandemic motivated the
hood obesity and mental health could be analyzed by updatavailability of diverse open datasets and indicators [1], preing the available longitudinal clinical with data covering the
senting new opportunities (new sources for monitoring,
lockdown period, reusing existing data preparation techmodelling and forecasting the pandemic) and challenges (the
niques, or modifying them to allow the inclusion of new
need to correctly pre-process and integrate the data sources
countries and health systems and reusing existing data analytmade available).
ics and visualization tools (on the period of interest). In the
In the MIDAS platform the GYDRA Big data preparation
case of childhood obesity, these could focus on body mass
tool (renamed from its initial in-memory processing version
index (BMI) and epidemiological analysis of new diabetes
TAQIH) [30] has been developed for the preparation, ingescases, covering person (gender, age-group), time (year) and
tion and loading of the selected datasets. GYDRA has two
location dimension (trust level and primary care unit level).
main aspects: (i) an easy to use and interactive web-based
Furthermore, the tool could be extended by plugging and
interface (mimicking traditional data quality assessment and
ingesting new data sources such as physical activity captured
improvement flow) allowing non-technical users to use it; and
by wearables, nutrition-related periodic questionnaires, or by
(ii) data synchronization functionality to allow data owners
alternatively adding grocery shopping aggregated data that,
and policy-makers to iteratively prepare and automatically
once integrated to comparable aggregation levels and des--
deploy the prepared data to the analytics platform (relying on
cribed in metadata, could be analyzed side-by-side with curApache Hive technology, and a defined metadata approach for
rent visualizations or further analyzed using statistical or
the platform).
machine learning technique.
Within GYDRA's data preparation user interface items are
placed from left to right following the usual iterative pipeline
Ingesting Useful Open Data Sources
in exploratory data analysis. The "General Stats and Features"
The MIDAS platform includes heterogeneous datasets prepared
GYDRA sections provide global and detailed views of the data
and deployed in different pilot site locations, addressing their
content, distribution and quality. The "Missing Values" section
own specific challenges. These include city and government
deals with the completeness of data. The "Correlations" section
generated controlled datasets (i.e. health and social care data
presents the correlations amongst variables, to help identify
exports mainly at individual person level) and government
possible redundancies amongst variables or incoherent data.
open data (aggregated data) on air and water quality, national
Finally, the "Outliers" section identiﬁes outliers for each varistatistics (e.g. deprivation, education level or unemployment
able. Based on the insights identified during this analysis, a
level per municipality), or city planning. These data sources are
transformation pipeline can be configured to drop features and
selected by each user to address various priority health policy
observations, handle missing values and outliers, or define operquestions across each pilot region.
ations to create new features from existing features or to
The integration of different controlled public health data
change specific values. Within the MIDAS project each dataset
sources containing individual level data for the MIDAS pilot
from each pilot site has required a different preparation recipe.
cases was completed by the data owners prior to loading
However, common tasks for each dataset include checking the
these into the MIDAS platform, while the linking identifiers
number of columns per row (to avoid issues with separators
and datasets provided to the consortium were agreed to be
being present in the content), merging data exported in
provided on an anonymous basis. A different instance of the
chunks, format changing (e.g. for date fields loading), recoding
MIDAS platform was securely hosted at each policy site, to
of categorical values (after defining integration mappings),
ensure the data owner retained control over the data being
dropping features with few occurrences, dropping some meanloaded. These heterogeneous datasets have been used to proingless outlying occurrences and creating new tables for specifvide combined solutions in different sites, combining data at
ic analysis.
aggregated and individual level, by providing mappings at
The metadata generated by the data synchronization funcagreed location aggregation levels. Applying this process to
tionality introduced above describes the data organization after
the context of COVID-19, it is easy to map and analyze the

NOVEMBER 2020 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - November 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2020