Editor’s note: Today’s blog post is from Alberto Asuero, CTO of CARTO,  the location intelligence platform. Today he shares more details about the source of the data for advanced data visualizations created with Google Maps Platform and deck.gl and how the CARTO platform enables this workflow.

During Google Cloud Next in October, the Google team announced the newest release of the deck.gl visualization library, thanks to a collaboration with our geospatial company CARTO and the vis.gl Technical Steering Committee (TSC). The deck.gl release includes a deep integration with the new WebGL-powered features in the Maps JavaScript API that allows deck.gl to render 2D and 3D visualizations directly on the Google basemap. 

Our team built an example app that visualizes a variety of data sources that show the potential for electrification of truck fleets in Texas. This app showcases the different types of advanced data visualizations that can be created with Google Maps Platform and deck.gl. Today, I want to share more details about the source of the data for these visualizations and how the CARTO platform enables this workflow. 

Google Cloud provides a strong serverless data warehousing solution, BigQuery, with support for geospatial queries. When you are dealing with spatial data, creating maps to explore and visualize these datasets is an important and common need. The CARTO Spatial Extension for BigQuery provides an easy way to create connections to the data warehouse, design a map with data coming from BigQuery tables, and then add these visualizations to a web app using deck.gl.

Making a simple map

To create a simple map using the CARTO platform, you can sign up for a trial account. Once you have signed in, you can set up a connection to your BigQuery instance using a service account. Then, you can go to the Data Explorer and browse the available datasets to find the table you want to use as the datasource in your map. For more information, check out the CARTO documentation.

To create a visualization of power transmission lines in Texas, you can start with the Texas state boundary to provide some context. In the Data Explorer, you can preview the table and click the “Create map” button in the top-right corner to start designing your visualization.

Using the CARTO Builder map making tool, select one of the available Google vector basemap styles and customize the layer style.

You can visualize tables and the results from queries executed in the data warehouse, which is a powerful feature because you can also execute spatial analysis functions using SQL, including those from the CARTO Analytics Toolbox. In this case, you can intersect the lines in the table containing all the U.S. transmission lines within the Texas boundary. Click on the “Add source from…” button and select the “Custom Query (SQL)” option to add the following query:

SELECT * 
FROM cartobq.nexus_demo.transmission_lines
WHERE ST_INTERSECTS(
  geometry, 
  (SELECT geom FROM cartobq.nexus_demo.texas_boundary_simplified)
);

Click the “Run” button, and the query is executed in BigQuery. The results are sent back to the Builder tool. Perform some style customizations in the new layer, and your map is ready.

Before adding the map to the Google Maps Platform application, you’ll need to make it public. Click on the “Share” button and select the “Developers” tab to copy the map ID.

Now, you can add the visualization into your Google Maps Platform application, which is as easy as adding these four lines of code:

const cartoMapId = 'b502bf53-877d-4e89-b5ad-71982cac431d';
deck.carto.fetchMap({cartoMapId}).then(({layers}) => {
  const overlay = new deck.GoogleMapsOverlay({layers});
  overlay.setMap(map);
});

You can use the map ID copied from CARTO Builder to call the fetchMap function. This function connects to the platform and retrieves all the information needed for the visualization, including a collection of deck.gl layers with all the styling properties you’ve specified. Create an instance of the deck.gl  GoogleMapsOverlay with this collection of layers and add it to the map.

You can see the full example in this fiddle.

Visualizing very large datasets

One of the main features of BigQuery is the ability to scale processing to massive datasets. With the CARTO platform, you can also visualize very large datasets using tilesets, an optimized data structure containing pre-generated vector tiles for fast visualization. Tilesets are generated within BigQuery using the Analytics Toolbox functions in a parallelized process that can handle billions of points.

For example, you can create a visualization using tilesets with the whole dataset of transmission lines for the U.S., more than 100MB of geometries. 

The issue with these large datasets is that they do not fit in memory all at once, so you need to split them into tiles for them to be rendered progressively. CARTO takes care of this, allowing you to create tilesets directly in BigQuery or dynamically generate them on the fly.

This method for data loading in maps can scale as much as needed; for example, take a look at this 17 billion point visualization of vessel data.

What about live data?

BigQuery supports streaming data that is continuously updated. In these scenarios, you want to be able to update your visualization at regular intervals, as the data changes. It’s easy to update this visualization using deck.gl. You just need to set the autoRefresh parameter to true when fetching the map and specify the function you want to execute when new data is downloaded:

const {layers} = await deck.carto.fetchMap({
  cartoMapId,
  autoRefresh: true,
  onNewData: (parsedMap) => { … }
});

You can add points to a table with an INSERT function on the BigQuery console and see the data updated on the map in real time.

Going further

In addition to the simple ways to create visualizations shown above, deck.gl has the flexibility to create a wide variety of visualizations. The CARTO platform provides you with the functionality to access data from your data warehouse and create these data visualizations with advanced cartographic capabilities, but you can extend it and go beyond that using any of the advanced visualizations available in the deck.gl layer catalog.

There are two additional options that give you more control over the deck.gl code. The first one is to use the CartoLayer directly without fetchMap. You’ll need to indicate the connection to use from the CARTO platform and the data source type and name or query. Then we can specify the styling properties.

const overlay = new deck.GoogleMapsOverlay({
  layers: [
    new deck.carto.CartoLayer({
      connection: 'bqconn',
      type: deck.carto.MAP_TYPES.TABLE,
      data: `cartobq.public_account.retail_stores`,
      getFillColor: [238, 77, 90],
      pointRadiusMinPixels: 6,
    }),
  ],
});

The second option is to use the fetchLayerData function that allows you to have more control over the format used for data transfer between BigQuery and your application and can be used with advanced visualizations that require an specific data format like ArcLayer, H3HexagonLayer or TripsLayer.

deck.carto.fetchLayerData({
  type: deck.carto.MAP_TYPES.TABLE,
  source: `cartobq.geo_for_good_meetup.texas_pop_h3`,
  connection: 'bqconn',
  format: deck.carto.FORMATS.JSON,
  credentials: {
     accessToken: 'eyJhbGciOiJIUzI1NiJ9.eyJhIjoiYWNfbHFlM3p3Z3UiLCJqdGkiOiI1YjI0OWE2ZCJ9.Y7zB30NJFzq5fPv8W5nkoH5lPXFWQP0uywDtqUg8y8c'
   }
 }).then(({data}) => {
  const layers= [
    new deck.H3HexagonLayer({
      id: 'h3-hexagon-layer',
      data,
      extruded: true,
      getHexagon: d => d.h3,
      getFillColor: [182, 0, 119, 150],
      getElevation: d => d.pop,
      elevationScale: 2.5,
      parameters: {
        blendFunc: [luma.GL.SRC_ALPHA, luma.GL.DST_ALPHA],
        blendEquation: luma.GL.FUNC_ADD
      }
    })
  ];
  const overlay = new deck.GoogleMapsOverlay({layers});
  overlay.setMap(map);
 });

For complete code using both options, take a look at these examples.

Learn more

You can access demos and documentation on the deck.gl docs website and the CARTO Documentation Center. If you have questions, you can ping the CARTO team on the CARTO Users Slack workspace. 

For more information on Google Maps Platform, visit the Google Maps Platform website.