Personal tools

Contact Us 24/7 > 1 866.SIX FEET
Sections

Skip to content. | Skip to navigation

Home > Blog > Spectacular SaltConf 2017

Blog

Banner image

Spectacular SaltConf 2017

written by Calvin Hendryx-Parker on Tuesday November 14, 2017

The Salt community gathered for SaltConf in Salt Lake City this year at the Salt Palace. I’m not sure I’ve ever written the word Salt so many times in one sentence before, but I like the ring of it. This was my first year attending the event and it was a blast. I’ve been around the Python community for years, and Salt definitely gets community and events done right.

Keynotes

Each morning kicked off with a set of keynotes from major partners in the Salt ecosystem, such as Adobe, Suse Linux, Bloomberg and Intuit.

A few notable announcements:

  • Intuit went from ink on paper to launching 30k Salt minions in just 6 months.

  • SaltStack announced a new Slack channel for the community. You can get an invite by going to the inviter and joining in the conversation.
  • SaltStack is also looking to double their engineering headcount over the next 12 months.
  • Bloomberg, who won one of the new Partner of the year awards, shared how they have written custom reactors to accept keys automatically based on external verification sources.

Take-Aways from Key SaltConf Sessions

GitFS and Branches

Gareth Greenaway (SaltStack) covered some great tools for Salt developers and users to streamline their workflow while using Salt.

He published an example project that uses GitFS and branches for environments. It is up on GitHub at https://github.com/garethgreenaway/saltconf17

Using the GitHub repo, you can try out the following:

  • versioned salt states
  • salt environments == project branches
  • Salt API + Reactors + Github Webhooks
  • SLS files can be straight python (good for reactors)

Authors of SaltStack components and states will appreciate the following tools mentioned in his presentation:

Custom Salt Modules

C.R. Oldham (SaltStack) went over creating custom Salt modules. The code from his presentation is also on GitHub at https://github.com/cro/saltconf17.

Interestingly, when building your modules, __virtual__ is not required, but is recommended, and you can use saltutil.sync_all to bring custom modules into the Salt cache.

C.R. also showed off one of his tools for automating his workflow by watching changes to states etc., and running them in real time by creating a quick shell script called sdc to call the states locally like this (aka the Salt Development Command):

#!/bin/bash
salt-call --local "$@"

You can combine that with the watchdog Python package to monitor the files in a directory and run them as they are changed.

Network Automation and Edge Device Control

C.R. Oldham (SaltStack) also gave a great talk on using salt-proxy to control devices that can’t run a minion.

Here is the code from the presentation: https://github.com/cro/saltconf17/tree/master/SaltStack_Network_Edge_Device

There is a sample REST endpoint code here that was used in the presentation: https://github.com/saltstack/salt-contrib

As a quick note, one process per controled device each uses about 40MB RAM and each device currently requires its own proxy process.

Powering a Next-Gen NOC and Self-Service Portal with SaltStack Enterprise

Stephan Looney and Tyler Jones (Sterling Talent Solutions) showed how they have greatly reduced the time needed to patch their systems. By leveraging Salt’s power with the nice easy UI of SaltStack Enterprise (SSE), they were able to empower other administrators (who had no idea what a state, pillar or orchestration were) to be able to perform common functions, such as patching. Essentially, it’s possible to fire custom events to run orchestrate via the GUI and let SaltStack do all of the heavy lifting. No more weekend patching party conference bridges!

Grokking Salt Arch and Design Paradigms

It is no surprise that Tom Hatch (SaltStack) loves plugins: they are, along with async, a core design principle of SaltStack.

During the presentation, Tom went over many of the core architecture pieces that make up Salt. He used this nice graphic to represent the inner workings/interactions:

One interesting new component of Salt that I’ve not tried is Thorium. It is a flow-based programming concept to do things that reactors alone can’t. Thorium allow you to watch for trends and react based on thresholds.

Docker Container Orchestration

This was probably one of my favorite sessions of the conference. Charles Ruffino (SaltStack) demonstrated building a serverless AWS Lambda-like service using Salt’s event bus and Docker. It combines reactors and the docker event engine to spin up and run functions in containers.

You can check out the demo from GitHub at https://github.com/SaltyCharles/saltconf/tree/master/17/serverless

Automagic Security Hardening

Benjamin Allen (BlueTarp Financial) introduced the idea of SecDevOps to the crowd and how they implement it at their organization using the following components:

  • SaltStack (Config Management, Policy Enforcer)
  • Packer (agnostic image builder)
  • Jenkins (Pipeline orchestration tool)
  • OSCAP & Nikto (Opensource auditing and scanning software)

One interesting fact is that, in 85% of the time, breaches are caused by not patching software. At BlueTarp Financial, they do not patch running systems: they torch the old ones and replace them with fresh ones from their newly built images.

Best Practices for Enterprise-Scale SaltStack

Alexander Thaller (trivago) showcased a scheme that was inspired by Puppet’s “Roles and Profiles” for managing configs across their environments and applications.

The main components are states that define:

  • Roles
  • Profiles
  • Formulas

With these building blocks in place, it’s possible to use Salt grains to assign minions to various environments and roles for applications. With profiles, formulas can be combined together and included in multiple roles as reusable components.

Alexander’s Salt pro-tip: you can move your formulas around and the file path won't break if you include files using this pattern:

salt['file.join'](tpldir,"map.jinja")

Alexander has built some cool utilities for use in Salt such as salt-compressor to minimize output noise.

He also created a saltstack_helper that looked like it used Graphviz to provide graphic visualization of the salt state dependancies.

Debugging Salt

Nice quick tips from Megan Wilhite (SaltStack) on debugging Salt:

(1) Install setproctitle for extra awesomeness on the master. It will now change the process title to show what function the Python process is performing.

(2) Use the following flags to get more legible info when debugging master and minion runs:

$ salt-call -ldebug
$ salt-run state.event pretty=True

(3) For Python debugging, here are some great options for diving deeper:

  • PDB/PUDB
  • remote-pdb
  • log.error()
  • print()

(4) Inside Jinja templates, all of the data available can be viewed using a snippet that shows the dictionary of the full context from the viewpoint of the template: {% do salt.log.error(show_full_context()) %}

Testing Salt States

Daniel Wallace (SaltStack) showcased using Kitchen Salt for automated testing of salt states. Kitchen Salt uses Ruby and Bundler to set up a test runner for building and asserting that your states run correctly.

To use Kitchen Salt, you configure a driver, which supports lots of platforms such as FreeBSD, most Linux distributions and Windows. You then set it up to run your suites, and you can pass custom bootstrap options to SaltStack to it matches your current process of installation.

From a Pythonista standpoint, you can use pytest to build your tests.

Daniel’s pro-tip: use things like the ssh driver against clones of prod machines to refine your states before applying them.

Conference Closing

The last session of the conference was a Q&A with Tom Hatch. The crowd was able to ask Tom just about anything they wanted. The questions and stories told ranged from how Salt got its name (from a scene in The Lord of the Rings) to a Star Wars face-off between Leia and Padmé. I asked about Salt’s support of the fileserver environments and if it is really used in production by others: it sounds like using grains might be a better option.

SaltConf 2018

I’m already looking forward to next year’s conference which will be September 10-14th, 2018 in Salt Lake City. Let me know if you’re thinking about going too!

 
Posted by William Cannon on Nov 15, 2017 09:16 AM
A new salt state testing tool was unveiled at SaltConf17 - saltcheck. Saltcheck is a custom salt execution module that was accepted into the salt open develop branch. This makes testing the logic of salt states Very easy. Here's a link to the presentation. https://docs.google.com/presentation/d/1zXbcIRPz7VC4p3gXxA_DS2A3_gHXdDr_jDo5dTYsFVQ/edit?usp=sharing full disclosure: I'm the author of saltcheck
Posted by calvin on Nov 15, 2017 09:28 AM
That looks awesome, I’m going to have to check that out. Given the tracks, I couldn’t make it to all of the talks, but I wish I could have. I’m looking forward to the videos being put up.
Add comment

You can add a comment by filling out the form below. Plain text formatting.

puzzle
Calvin Hendryx-Parker
Chief Technology Officer
Calvin's Recent Posts:
Spectacular SaltConf 2017 (11/14/2017)

Next Steps


Select a type of support:

Contact our sales team

First name:
Last name:
Email:
Phone Number:
Message:
Fight spam:
What is + ?
 
Call Us 1 866.SIX FEET
Sections