{
  "path": "/posts/2016/2016-03-25-presto-cluster",
  "site": "at://did:plc:mracrip6qu3vw46nbewg44sm/site.standard.publication/self",
  "tags": [
    "code",
    "presto"
  ],
  "$type": "site.standard.document",
  "title": "Creating a Presto Cluster",
  "updatedAt": "2016-03-25T17:19:00.000Z",
  "publishedAt": "2016-03-25T17:19:00.000Z",
  "textContent": "Creating a Presto Cluster\n\nI first came across Presto when researching data virtualization - the idea that all of your data can be integrated regardless of its format or storage location. One can use scripts or periodic jobs to mashup data or create regular reports from several independent sources. However, these methods don't scale well, especially when the queries change frequently or the data is ingested in realtime. Presto allows one to query a variety of data sources using SQL and presents the data in a standard table format, where it can be manipulated and JOINed like traditional relational data.\n\nIn Facebook's words:\n\n> Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.\n\nThere is a guide for how to download and setup Presto here. Additionally, there is a well documented [list of connectors][connectors_link] for querying a wide variety of data sources (including Redis and Kafka!). However, the documentation on how to setup a cluster (coordinator and workers) is a little sparse.\n\nThis guide assumes you have machines runnning Java 8 that can interact with each other over port 8080. Some of the documentation for installation from Facebook is repeated for the sake of having all the instructions in one place. Additionally, you'll need to have data sources to which Presto can connect. See the full list on the [connectors page][connectors_link].\n\nInstall Presto\n\nDownload Presto. These instructions are for version 0.142. You can find the most recent version and more deployment information here.\n\nDownload the CLI for the coordinator and name it presto\n\nCreate configure files\n\nCreate etc folder in presto-server-0.142 directory\nCreate config.properties, jvm.config, log.properties, and node.properties files.\n\nInstall uuid to generate a node.id\n\nnode.properties\n\njvm.config\n\nThis is the configuration for the coordinator:\n\nCoordinator config.properties\n\nThe above files can be configured as needed for your specific cluster. The options node-scheduler.include-coordinator=false and coordinator=true indicate that the node is the coordinator and tells the coordinator not to do any of the computation work itself and to use the workers. After configuring the coordinator, you need to configure workers to add some resources to the cluster. If you were to try and run a query now, you would see something like:\n\nFor the workers, you need to create all the same files as above, but the config.properties file needs to look similar to the one below.\n\nWorker config.properties\n\nNext we will create the catalog folder which tells Presto how to connect to various data sources. This folder goes inside the etc folder on all nodes of the cluster including the coordinator. An example folder structure might look like:\n\nConsult the [connectors page][connectors_link] on how to write theses properties files. It may help to version control this folder to make it more easily distributable.\n\nStart Presto\n\nNow that we've set up our coordinator and worker node(s), we can start the cluster. First SSH into and start the coordinator node:\n\nSSH into and start the worker node(s):\n\nOnce you start the workers, you can use the Presto CLI on the coordinator to ensure the workers are connected:\n\nNow you can use the Presto CLI on the coordinator to query data sources in the catalog using the Presto workers.\n\n[connectors_link]: https://prestodb.io/docs/current/connector.html",
  "canonicalUrl": "https://www.danielcorin.com/posts/2016/2016-03-25-presto-cluster"
}