title

Rene´s blog

Blogging about Programming, Security, Linux, Networking and Web Apps.

Howto move Saltstack tops and pillar contents to MongoDB

Detailed explanation on HOWTO move your Saltstack tops and pillar data to MongoDB


I’m a heavy user of Saltstack, on my home network i develop salt states and test new stuff and on my production servers i use the results of
my development at home.

My motiviation for this:

  • I have a dream of automated deployed LXD containers which you can manage with a web interface like Froxlor.
  • I want a database where i can easily modify contents with a script.

Prerequisites

  • Knowledge of Saltstack
  • Knowledge of MongoDB

Salt Modules in use

Here comes the step-by-step guide

1.) Install MongoDB somewhere and create some users

a.) Go to the MongoDB installation guide for install instructions.

b.) Create a superadmin user

Open a mongo shell:

mongo

And insert the following (replace the username and password!).

use admin;
db.createUser({ user: "<replace with your username>",
  pwd: "<replace with your cleartext password>",
  roles: [
    { role: "clusterAdmin", db: "admin" },
    { role: "userAdminAnyDatabase", db: "admin" },
    { role: "readWriteAnyDatabase", db: "admin" },
    { role: "dbAdminAnyDatabase", db: "admin" },
  ]
})
quit();

c.) Configure MongoDB to enforce authentication:

Add this to your /etc/mongd.conf:

security:
  authorization: enabled

And restart MongoDB:

service mongod restart

d.) Create a user and Database for your saltmaster:

Open a mongo shell and login

mongo -u <username from above> --authenticationDatabase admin -p
use saltstack;
db.createUser({ user: "saltmaster",
  pwd: "<replace with your cleartext password for the saltmaster user>",
  roles: [
    { role: "readWrite", db: "saltstack" },
  ]
})
quit();

2.) Configure your saltmaster to use salt_tops and salt_pillar with the MongoDB

a.) Open /etc/salt/master and insert

master_tops:
  mongo:
    id_field: _id
    collection: salt_tops

ext_pillar:
  - mongo: {collection: salt_pillar}

ext_pillar_first: false

#####   mongodb connection settings  #####
##########################################
mongo.db: saltstack
mongo.indexes: true
mongo.host: <your mongo host>
mongo.user: saltmaster
mongo.password: <your saltmaster mongo password>
mongo.port: 27017

You can also use the salt-formula, but you need the latest version with my PR

b.) Restart your salt-master

service salt-master restart

3.) Create some tops and pillars

I use robomongo for that, its a Desktop app with functionality like phpMyAdmin.
To convert my old YAML files to JSON i use: YAML to JSON.

a.) An example top in the collection salt_tops

{
    "_id" : "apu1d4.pcdummy.lan",
    "states" : [
        "roles.base.server",
        "roles.base.lxc",
        "bird",
        "softether.client"
    ],
    "environment" : "pcdummy"
}

b.) An example pillar entry in the collection salt_pillar

This uses my mongo include patch which you can optain from Salt PR #34566

{
    "_id" : "apu1d4.pcdummy.lan",
    "include" : [
        {
            "file" : "roles.base.server",
            "saltenv" : "pcdummy"
        },
        {
            "file" : "roles.base.lxc",
            "saltenv" : "pcdummy"
        },
        {
            "file" : "roles.base.sysctl_container_host",
            "saltenv" : "pcdummy"
        },
        {
            "file" : "roles.base.postfix-relayclient",
            "saltenv" : "pcdummy"
        }
    ],
    "grub" : {
        "lookup" : {
            "config" : {
                "manage" : [
                    "default_config"
                ]
            }
        },
        "default_config" : {
            "content" : "GRUB_DEFAULT=0\nGRUB_TIMEOUT=10\nGRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`\nGRUB_CMDLINE_LINUX_DEFAULT=\"quiet cgroup_enable=memory swapaccount=1\"\nGRUB_CMDLINE_LINUX=\"console=ttyS0,115200n8 earlyprint=ttyS0,115200n8\"\nGRUB_TERMINAL=serial\nGRUB_SERIAL_COMMAND=\"serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\"\n"
        }
    },
    "network" : {
        "hosts" : [
            {
                "name" : "apu1d4.pcdummy.lan",
                "ip" : "fd57:c87d:f1ee:ee00::1"
            }
        ],
        "resolver" : {
            "domain" : "pcdummy.lan",
            "search" : [
                "pcdummy.lan"
            ],
            "nameservers" : [
                "fd57:c87d:f1ee:ee00:f::18"
            ]
        }
    },
    "lxc" : {
        "default_conf" : [
            {
                "lxc.network.type" : "veth"
            },
            {
                "lxc.network.link" : "apubr0"
            },
            {
                "lxc.network.flags" : "up"
            },
            {
                "lxc.network.hwaddr" : "00:16:3e:02:xx:xx"
            }
        ],
        "users" : {
            "lxd" : {
                "interfaces" : {
                    "apubr0" : {
                        "type" : "veth",
                        "count" : 100
                    }
                }
            }
        }
    },
    "softether" : {
        "lookup" : {
            "client_svc_onboot" : true
        },
        "interface" : {
            "enabled" : true,
            "name" : "vpn_gw0",
            "ipv4address" : "10.171.104.160",
            "ipv4netmask" : "255.255.0.0",
            "ipv6enabled" : true,
            "ipv6address" : "fd57:c87d:f1ee:f003::ee00:1",
            "ipv6netmask" : 64
        }
    },
    "bird" : {
        "bird_cfg" : "log syslog { info, remote, warning, error, auth, fatal, bug };\nlog stderr all;\n\nrouter id 10.171.104.160;\n\nprotocol kernel {\n        learn;\n        persist;\n        scan time 20;\n        import all;\n        export all;\n}\n\nprotocol device {\n        scan time 10;           # Scan interfaces every 10 seconds\n}\n\nprotocol ospf main {\n        import all;\n        export all;\n\n        area 0.0.0.0 {\n                interface \"apubr0\";\n                interface \"vpn_gw0\";\n        };\n}\n",
        "bird6_cfg" : "log syslog { info, remote, warning, error, auth, fatal, bug };\nlog stderr all;\n\nrouter id 10.171.104.160;\n\nfunction is_default() { return net ~ [ ::/0 ]; }\n\nprotocol kernel {\n        learn;\n        persist;\n        scan time 20;\n        import all;\n        export all;\n}\n\nprotocol device {\n        scan time 10;           # Scan interfaces every 10 seconds\n}\n\nprotocol ospf main {\n        import all;\n        export filter {\n                if (is_default()) then reject;\n                accept;\n        };\n\n        area 0 {\n                interface \"apubr0\";\n                interface \"vpn_gw0\";\n        };\n}\n\nprotocol radv {\n    interface \"apubr0\";\n    prefix fd57:c87d:f1ee:ee00::/64;\n    prefix 2001:470:b718:ee00::/64;\n\n    rdnss fd57:c87d:f1ee:ee00:f::18;\n\n    dnssl {\n      domain \"pcdummy.lan\";\n    };\n}\n"
    }
}

4.) Check if your tops and pillar.items are right

On the saltmaster

a.) For the tops

salt apu1d4.pcdummy.lan state.show_top

b.) For the pillar

salt apu1d4.pcdummy.lan pillar.items

5.) Leave a comment about this HOWTO

Any suggestions? Or did it help you? Please leave a comment.