[chef] Re: Re: Re: Simultaneous node edits

[Maxime Brugidou < >]

I don't know about the server side but wouldn't it be better to have this feature directly usable from the Chef gem? Having something like node.safe_save in addition to node.save instead of only implementing this for the knife node edit command. This could help other gems use the feature.

And should we do a safe_save or a save at the end of a Chef run? Because that's where we get most of the concurrent edits: a Chef run happening while the node gets edited, and doing a safe_save from knife won't solve this. But do we want to make the Chef run fail for that? Or retry?

On Jul 8, 2015 10:18 PM, "Rafał Trójniak" < "> > wrote:

On Tue, 7 Jul 2015 14:56:17 -0700
Noah Kantrowitz < "> > wrote:

>
> On Jul 7, 2015, at 2:33 PM, Rafał Trójniak < "> > wrote:
>
> > On Mon, 06 Jul 2015 10:20:15 -0700
> > Lamont Granquist < "> > wrote:
> >
> >>
> >>
> >> On 07/01/2015 11:40 PM, Maxime Brugidou wrote:
> >>> I would like to stop using Chef nodes as file but use the new
> >>> chefDK provision command with a special driver that would "pick" a
> >>> node from the firstboot pool (so basically my "cloud" provider is
> >>> the pool of firstboot nodes in Chef). Without dealing with
> >>> concurrent access to Chef provision, this seem doable: to allocate
> >>> a node I can "tag" a firstboot node and delete it once the machine
> >>> is ready.
> >>>
> >>> But how to do this with concurrent access? It seems almost
> >>> impossible. And the way things are going with Policy files will
> >>> tend towards a separate git repo and provision cookbook per
> >>> policy, all sharing the same pool of firstboot nodes (for now I
> >>> don't use Policy files).
> >>>
> >>> I wish I could have a way to "lock" a node or something like that.
> >>>
> >>>
> >> The way to do this is to make sure only one agent on your network
> >> can move the node between states.  A simple design would be to
> >> have the node responsible for publishing that its done with
> >> firstboot by tagging itself and then the node.save at the end
> >> publishes the write.  Then write a simple web endpoint which is
> >> your API to 'allocate' a new firstboot'ed node.  By centralizing
> >> it you don't have to worry about race conditions between multiple
> >> clients all trying to get the same node at the same time.  You can
> >> then write command line tools that talk to the endpoint you wrote
> >> to get a node, rather than wanting a distributed lock that the CLI
> >> commands can grab on the node object itself.  If you've already
> >> got etcd or something similar that you're using internally you
> >> could probably use that instead.
> >>
> >
> > Hello
> >
> > Had anyone analysed lock-free and optimistic approach by using
> > 'If-Match:' HTTP header on the write stage ?
> >
> > The scenario would look like :
> > - Every object (node, role, environment) would have some token
> >  (could be timestamp, or any other value changed on each edit)
> > - When the user invokes 'knife node edit' the version is sent to
> > client (possibly in HTTP Header)
> > - When the user edits the object, the value is stored somewhere
> > - When the user sends write API call to the server, it sends
> > 'If-Match' header with value received in first call
> >  - If the token matches the old one - the object is updated
> >  - If the token does not match the old one - the update is rejected.
> >
> > That won't solve all the problems, but it will fix many of them
> > with (i suppose) less work and changes. Such behaviour would also be
> > non-braking change.
>
> This was discussed way back at the first community summit, but no one
> has written the code. I'm sure it would be accepted if someone sent
> in a patch though.
>
> --Noah

Hello Noah,

Thanks for the information. Just spent some time on the analysis on how
to achieve that.
- On the knife side
  - Extracting 'Etag' information from the response on the Chef::REST
  - Storing it during the edit (probably in Chef::Knife::NodeEditor
  - Updating node from the NodeEditor class instead of Node class
- On the chef-server side
  - I have completely no Idea where to start analysis from :/ I hadn't
    touched Erlang in few months.
  - We would generally require :
    - Generating Etag (dynamically or storing it somewhere) for the
      object (like node, environment...)
    - Sending Etag in HTTP Response header when requested by client
    - On Modification request, detecting 'If-Match' header
      - Comparing received value with current Etag, accepting/rejecting
        update

I hope that pattern will work for all objects (roles, nodes,
environments..)
Also this should work not generate braking change with older
chef-server/knife implementation.

Is there any way someone could help me with chef-server side ?

Regards,
--
Rafał Trójniak
WEB : http://trojniak.net/
: ">
Jid : ">
GPG  key-ID : 9A9A9E98
ABC8 83DF E717 6B76 CE49
BAFD 4F6F 854F 9A9A 9E98