More flexible create-module.py and contrib/wscript

Consider a project in contrib which wants this structure pulled from a single repo:

contrib/
    wscript
    project/
        .repo            // could be .git or .hg, for example
        main/
            wscript      // builds executable 'project' from main.cc and other.cc
            main.cc
            other.cc
        modules/
            moduleA/     // Contains normal module structure
              wscript
              model/
              ...
            moduleB/     // Contains normal module structure
        data/            // No source code here, so no wscript

One can't do this at the moment for two reasons:

*  contrib/wscript won't recognize the project directory because
   the top level does not contain a wscript file

*  Each "normal" module directory has to be created by hand,
   or created in src, then moved to its final location.


This patch provides two enhancements to enable this kind of project:

1. Enable the create-module.py script to create modules
   in the contrib directory, as well as the src directory.

2. Enable modules in the contrib directory to have arbitrary
   structure.

To accomplish these goals the patch modifies a couple of heuristics:

*  create-module.py accepts arbitrary paths, using the last path element
   as the name of the module (but see the question below).

*  contrib/wscript walks the directory tree, looking for directories
   containing wscript files. When found, add that directory to the
   list of contrib modules to process.  It is not an error for a tree
   under contrib not to contain any wscript files; this tree is simply
   ignored.  (The data directory above is an example.)

Since create-module.py is more general, this patch moves it to the utils
directory.

Question:  Should we restrict to just module paths matching src/new-module and
contrib/new-module?  The first restriction might be useful. The
second conflicts with allowing contrib modules to have arbitrary
structure.

[Tom Henderson, 2018-04-25 17:22:55]

(this is mainly an initial comment to check whether it will post to
ns-3-reviews; initial issue did not post there for some reason)

Is there a need perhaps to have both 'create-module.py' and 'create-project.py'?

[Peter Barnes, 2018-04-25 17:34:43]

On 2018/04/25 17:22:55, Tom Henderson wrote:
> Is there a need perhaps to have both 'create-module.py' and
'create-project.py'?

What would create-project.py do differently from create-module.py?

[Tom Henderson, 2018-04-25 17:42:43]

On 2018/04/25 17:34:43, Peter Barnes wrote:
> On 2018/04/25 17:22:55, Tom Henderson wrote:
> > Is there a need perhaps to have both 'create-module.py' and
> 'create-project.py'?
> 
> What would create-project.py do differently from create-module.py?

Create a module like it does presently (if there is no need for the hierarchical
structure).

I was not thinking about creating this kind of hierarchy recently; rather was
thinking of splitting each module into its own repo and using submodules to tie
together a flat structure.

I see the utility of having the project grouping of modules, so perhaps (instead
of my suggestion above for keeping create-module.py) we consider to use the
project structure everywhere.

[Peter Barnes, 2018-05-24 20:09:41]

Fixed test issue in listing available test.

[Peter Barnes, 2018-08-06 23:13:49]

@Tom:  I don't understand your last comment re: project grouping and project
structure...

Are there any other comments on this patch?
Peter

[Tom Henderson, 2018-08-12 22:48:11]

On 2018/08/06 23:13:49, Peter Barnes wrote:
> @Tom:  I don't understand your last comment re: project grouping and project
> structure...

My comment was about whether we want to encourage or allow that modules can be
standalone or part of a 'project' (that is, whether we want to encourage a
uniform usage of a project structure; e.g. 

utils/create-project.py contrib/projectname moduleNameA moduleNameB moduleNameC

would yield

contrib/
        projectname/
                    moduleNameA
                    moduleNameB
                    moduleNameC

and do not continue with the standalone 'create-module.py'

However, looking at this issue again today, I think your current proposal is
fine; if someone wants to create the 'project/' and 'data' structure themselves,
and add modules to it as they go, they can call your (revised) create-module.py
after creating the project structure that they want, or if they instead just
want something simple, they can use create-module.py the same way as before.

[Peter Barnes, 2018-08-13 17:12:47]

On 2018/08/12 22:48:11, Tom Henderson wrote:
> 
> utils/create-project.py contrib/projectname moduleNameA moduleNameB
moduleNameC
> 
> would yield
> 
> contrib/
>         projectname/
>                     moduleNameA
>                     moduleNameB
>                     moduleNameC
> 
> and do not continue with the standalone 'create-module.py'

Ahh, now I get it.  An alternative to supporting create-project.py *and*
create-module.py might be:

utils/create-module.py --project projectname moduleA moduleB moduleC

One script to rule them all.

Do projects only make sense in contrib (as I've implicitly assumed in the
command above)?  Or should we the script be agnostic?

[Peter Barnes, 2018-09-11 00:01:43]

> utils/create-module.py --project projectname moduleA moduleB moduleC
> 
> One script to rule them all.
> 
> Do projects only make sense in contrib (as I've implicitly assumed in the
> command above)?  Or should we the script be agnostic?

This version creates multiple modules at once.  If the project is given, the
modules are all created within that project. If not project is given it assumes
'src'. The project path must begin with 'src' or 'contrib', and may contain an
arbitrary path after that.

Module and project names may only contain [a-zA-Z0-9_-]

[Tom Henderson, 2018-10-08 19:05:11]

I support merging this, although my testing of it today suggests it may need
tweaking.

    $ python utils/create-module.py mymodule
    Project src already exists
    Creating module src/mymodule

    Successfully created new modules
    Run './waf configure' to include them in the build

Do we make 'src' or 'contrib' the default?  I might suggest the latter.

Do we require consistency that 'src' or 'contrib' must be explicitly specified? 
The implicit 'src' project causes some problems; e.g.:

This does not work as I would expect:

    $ python utils/create-module.py src/mysecondmodule
    Project src already exists
    Creating module src/src/mysecondmodule
    Traceback (most recent call last):
      File "utils/create-module.py", line 484, in <module>
        sys.exit(main(sys.argv))
      File "utils/create-module.py", line 473, in main
        ok = make_module(modpath, modname)
      File "utils/create-module.py", line 394, in make_module
    os.mkdir(moduledir)
    OSError: [Errno 2] No such file or directory: 'src/src/mysecondmodule'

This also doesn't work as documented:

    $ python utils/create-module.py contrib/mycontribmodule
    Project src already exists
    Creating module src/contrib/mycontribmodule
    Traceback (most recent call last):
      File "utils/create-module.py", line 484, in <module>
        sys.exit(main(sys.argv))
      File "utils/create-module.py", line 473, in main
        ok = make_module(modpath, modname)
      File "src/create-module.py", line 394, in make_module
        os.mkdir(moduledir)
    OSError: [Errno 2] No such file or directory: 'src/contrib/mycontribmodule'

Also, a similar error occurs if the user specifies the wrong project, such as
mistakenly trying to add the module to the 'utils' directory.

    python utils/create-module.py utils/myutilsmodule
    ...
    OSError: [Errno 2] No such file or directory: 'src/utils/myutilsmodule'

[Peter Barnes, 2018-10-08 19:25:46]

On 2018/10/08 19:05:11, Tom Henderson wrote:
> Do we make 'src' or 'contrib' the default?  I might suggest the latter.
> 
> The implicit 'src' project causes some problems; e.g.:
> 
> This does not work as I would expect:
> 
> This also doesn't work as documented:
> 
> Also, a similar error occurs if the user specifies the wrong project, such as

Thanks for testing and providing failing examples.  We'll do some more work and
post a revised patch.
Peter

[Peter Barnes, 2018-11-12 19:16:39]

Also need to revise the file templates to refer to contrib module files by the
correct path (not 'src/foo/...')