[chef-dev] Re: Re: Re: Re: local file copy resource?

[Daniel DeLeo < >]

On Monday, December 10, 2012 at 6:57 PM, Jesse Campbell wrote:

I agree with you that there needs to be a better understanding of the end goal.

My end goal is this:
1. I want to be able to pick up a text or binary file from anywhere (local, cookbook, remote, text string) and drop it somewhere, and have it just work.
2. I want to be able to pick up a text file from anywhere (local, cookbook, remote, text string), feed it through template rendering, and drop it somewhere, and have it just work.
3. Not break current functionality (existing resources should continue to exist and should function the same way)

Right now I can do many of these but I can't:
- copy/move a local binary file without an execute resource shelling out to cp, or rubycode running FileUtils.cp
- fetch a remote file from anything except for non-authenticated HTTP/HTTPS without shelling out to wget/curl
- feed a remote file through template rendering, without having first a remote_file to drop it into tmp, then a template with local option to pick it back up

This can all be accomplished without changing the way chef models things, which is what I'm concerned about. These are all reasonable improvements.

 

Additionally there are three different resources that all just make a file.

I do not see a connection with directory, so I would not put directory or remote directory in with file (though puppet does exactly this) I'm not going into it here, but why the f does remote_directory pull things from a cookbook, and remote_file pull things from the web? who named these things? okay, rant over.

There's a handful of places in the current design that are the result of seeking local maxima or plain historical accident. Cookbook file vs. remote directory is one of them.

Fixing inconsistencies like this is probably the next big improvement we can make in non-workflow user experience in Chef.

If I combine remote_file, template, cookbook_file, and file all together, at least the way I think of them, we end up with the following attributes:
path/name: the target location
diff: i don't know what this does, but its in the base file
backup: number of backups to keep for new file
content: text string to use for file content
local: boolean to say whether the file is local instead of cookbook
remote: boolean to say whether the file is remote instead of cookbook
source: accepts path on filesystem when local, path inside cookbook/files, path inside cookbook/templates when a template, URL when remote; causes content to be ignored
cookbook: name of the cookbook when source is specified, local is false, remote is false, ignored otherwise
template: boolean to say whether to perform template rendering
variables: the template variables hash, ignored unless template is true

These are added to base file: local, remote and template booleans, source, cookbook, and variables.
Combinations that don't make sense:
1. specifying local or remote, then specifying cookbook **you can already do this in template**
2. specifying source, then specifying content **you can already do this in template, cookbook_file, and remote_file**
3. specifying variables, but not specifying template **this one is new, but we could throw an error**
4. specifying local and remote at the same time **this one is new and should throw an error**

Any other combination should work.

This is a case of historical accident again: the validation "framework" we have does not operate on whole resources, only individual fields. So there's no consistent way to express that a certain parameter to a resource is required, or that two parameters are exclusive. If we're going to change the model such that these things are much more likely to occur, then we ought to invest in leveling-up our validation code so we can catch these things.

You are correct that the implementations are currently different, but why should they be? this is not a complex problem, yet it is handled in a very complex way.

The way things are factored currently isn't ideal, but, for the most part, all the code that can be shared, is shared. I was simply talking about the bits that are irreducible, e.g. template rendering for templates, file fetch via Chef-protocol authenticated HTTP for cookbook file, etc.

 

remote file fetches should be using a ruby standard library to fetch a file, not the custom chef rest client.

This is orthogonal to the design and modeling of the various resources. What are the actual problems you want to solve here? FWIW, Chef does use rest-client in some places instead of Chef::REST, so if that fixes your problems you should be able to switch to that without a lot of effort.

 

template should be able to be applied to any generic thing, so have the provider stage every file first with deploy_tempfile, then if template=true apply template processing in place on the tempfile, then deploy.

How would you use this?

-Jesse

As I mentioned above, I think the biggest issue we have in the chef model and DSL right now are little inconsistencies. What you're proposing here is something that could help make our model more consistent, but only if we apply it broadly. If we only apply it to this one set of resources, then we'll end up with more inconsistencies, since files would be managed by a do-everything file resource, whereas packages would still be gem_package, easy_install_package, dpkg_package, etc. So the core question for me is, "what are the tradeoffs of managing everything of a general type as a single resource vs. implementation-specific resources?" All of the bug fixes/improvements you've talked about can be implemented within the current implementation-specific resource model, so it's not necessary to get the functionality you need.

There are some features of the current implementation-specific resource model that I like:

* Exposing implementation makes things more obvious, for example, I can see a template resource and know I need to look at the corresponding template to see how the file gets its content.

* It's more succinct to express, e.g., `cookbook_file "path/to/file.conf"` can infer the source location and backend implementation

* The implementation is less complicated, since resources have less logic for dealing with different implementations, validations, etc.

You did say that you didn't plan to remove any of the existing implementation-specific resources, but there are costs to maintaining both models:

* Users will share cookbooks written with whichever style they prefer, so users will have to understand both to read or modify community cookbooks

* both approaches need to be documented, which leads to more confusing docs, since they have to explain multiple ways to accomplish one thing

* There will be extra code, with extra tests, which is more stuff to maintain and change when adding a feature or refactoring

Unless there's a compelling reason to maintain both indefinitely, I'd prefer to have one approach be the "blessed" one and then deprecate the other.

- Dan