Move app config to the ini as the single location.

Move app config to the ini as the single location.

There are several constant/setting values spread out across the app. This
starts an attempt to move it into the single ini file used for the
application. It changes the web app to read them during request time and for
other apps (jobs) to be able to easily load them from the current ini file.

The ini file defaults to charmworld.ini, however it respects the ENV variable
INI for changing that at run time (say to test a different db server, etc).

test.ini
--------
Add a test.ini that is a series of settings specific to the test runner. This
allows us to run without the debug settings from the development ini. It also
allows for different settings, such as lower cache times, a different db
connection, etc.

The makefile is updated to pass the test.ini in as the INI= environment
variable.

cached_view_config
------------------
The one issue is that the appliactions settings aren't available during
config.scan() so that we can get at them in order to set the http_cache
setting. Since it's really just duped all over we add a custom
cached_view_config that reads the http_cache value from the app's ini file.

The only trick then was to support having that value be different by a
specific offset. The cached_view_config supports a new param
http_cache_modifier which is a value that is multiplied by the cache value.
This allows us to keep from hard coding any cache values, yet support the
recent routes that use a HTTP_CACHE/2 value for their cache.

initialize_db
-------------
To setup the infrastructure for moving the datastore connection info out the
initialize_db is set back up in the models code. This is like the
initialize_sql method used in sql based pyramid apps. During main() the db
connections are setup and shared through the rest of the app. Currently, the
only datastore moved is the mongo connection and to prove it works the
jobs/config.py reads the mongo connection info from the new
charmworld.utils.get_ini response.

charmworld.utils.get_ini
-------------------------
Is a new helper to fetch the correct ini file and parse the data into a
dictionary. It attempts to keep the same api expected from the pyramid parsed
version. Tests are added for this.

As a side, in order to clean up the utils.py, tests for the pretty_timedelta
were added into the new test_utils module.

https://code.launchpad.net/~rharding/charmworld/move_constants/+merge/139039

(do not edit description out of merge proposal)

[rharding, 2012-12-10 18:06:18]

Please take a look.

[hazmat, 2012-12-11 12:57:57]

this looks good. one issue though is the testing structure. currently its using
a top level test package with module uncorrelated test module names. i'd prefer
to keep tests closer to the code (sub package tests) with test modules named
after the modules in question. it makes a bit easier to find the test for any
given piece of code. 

ie. given


M	 charmworld/tests/__init__.py	 View		1 chunk	+5 lines, -2 lines	 0 comments	
Download
A	 charmworld/tests/test_utils/__init__.py	 View		0 chunks	+-1 lines, --1 lines	
0 comments	 Download
A	 charmworld/tests/test_utils/test_formatters.py	 View		1 chunk	+30 lines, -0
lines	 0 comments	 Download
A	 charmworld/tests/test_utils/test_ini.py	 View		1 chunk	+26 lines, -0 lines	 0
comments	 Download
M	 charmworld/tests/test_views/test_mainviews.py


all of test_utils subpackage would just be in a module test_utils.py.

test_views would go to views/tests and s/test_mainviews.py/test_views.py (what's
a mainview ?) 

what do you think?

[rharding, 2012-12-11 13:25:31]

On 2012/12/11 12:57:57, hazmat wrote:
> this looks good. one issue though is the testing structure. currently its
using
> a top level test package with module uncorrelated test module names. i'd
prefer
> to keep tests closer to the code (sub package tests) with test modules named
> after the modules in question. it makes a bit easier to find the test for any
> given piece of code. 
> 
> ie. given
> 
> 
> M	 charmworld/tests/__init__.py	 View		1 chunk	+5 lines, -2 lines	 0 comments	
> Download
> A	 charmworld/tests/test_utils/__init__.py	 View		0 chunks	+-1 lines, --1
lines	
> 0 comments	 Download
> A	 charmworld/tests/test_utils/test_formatters.py	 View		1 chunk	+30 lines, -0
> lines	 0 comments	 Download
> A	 charmworld/tests/test_utils/test_ini.py	 View		1 chunk	+26 lines, -0 lines	
0
> comments	 Download
> M	 charmworld/tests/test_views/test_mainviews.py
> 
> 
> all of test_utils subpackage would just be in a module test_utils.py.
> 
> test_views would go to views/tests and s/test_mainviews.py/test_views.py
(what's
> a mainview ?) 
> 
> what do you think?

Definitely no problem relocating the tests into their modules. I personally
prefer more files with less code to make it easier to find the tests. I'd
suggest keeping the directories and just adding a tests module to views, utils,
etc if that's ok. 

Mainviews is definitely an admittingly poor name. I'll just dump those tests
back into the views/tests/__init__.py to match their location in
views/__init__.py. 

If it's ok, I'd like to land this with these changes and then have a follow up
that reorg's so that it's just clear diff of moved files vs this with a lot of
code changes.

[hazmat, 2012-12-11 13:30:42]

On 2012/12/11 13:25:31, rharding wrote:
> On 2012/12/11 12:57:57, hazmat wrote:
> > this looks good. one issue though is the testing structure. currently its
> using
> > a top level test package with module uncorrelated test module names. i'd
> prefer
> > to keep tests closer to the code (sub package tests) with test modules named
> > after the modules in question. it makes a bit easier to find the test for
any
> > given piece of code. 
> > 
> > ie. given
> > 
> > 
> > M	 charmworld/tests/__init__.py	 View		1 chunk	+5 lines, -2 lines	 0
comments	
> > Download
> > A	 charmworld/tests/test_utils/__init__.py	 View		0 chunks	+-1 lines, --1
> lines	
> > 0 comments	 Download
> > A	 charmworld/tests/test_utils/test_formatters.py	 View		1 chunk	+30 lines,
-0
> > lines	 0 comments	 Download
> > A	 charmworld/tests/test_utils/test_ini.py	 View		1 chunk	+26 lines, -0
lines	
> 0
> > comments	 Download
> > M	 charmworld/tests/test_views/test_mainviews.py
> > 
> > 
> > all of test_utils subpackage would just be in a module test_utils.py.
> > 
> > test_views would go to views/tests and s/test_mainviews.py/test_views.py
> (what's
> > a mainview ?) 
> > 
> > what do you think?
> 
> Definitely no problem relocating the tests into their modules. I personally
> prefer more files with less code to make it easier to find the tests. I'd
> suggest keeping the directories and just adding a tests module to views,
utils,
> etc if that's ok. 
> 
> Mainviews is definitely an admittingly poor name. I'll just dump those tests
> back into the views/tests/__init__.py to match their location in
> views/__init__.py. 
> 
> If it's ok, I'd like to land this with these changes and then have a follow up
> that reorg's so that it's just clear diff of moved files vs this with a lot of
> code changes.

sounds good re followup. re views/tests/__init__.py and views/__init__.py it
would be nice if we could extract those files down to minimal contents and place
into appropriate modules, package __init__ files tend to be a be a hidden as a
code location ie non obvious without inspection.

[rharding, 2012-12-11 13:32:11]

On 2012/12/11 13:30:42, hazmat wrote:
> On 2012/12/11 13:25:31, rharding wrote:
> > On 2012/12/11 12:57:57, hazmat wrote:
> > > this looks good. one issue though is the testing structure. currently its
> > using
> > > a top level test package with module uncorrelated test module names. i'd
> > prefer
> > > to keep tests closer to the code (sub package tests) with test modules
named
> > > after the modules in question. it makes a bit easier to find the test for
> any
> > > given piece of code. 
> > > 
> > > ie. given
> > > 
> > > 
> > > M	 charmworld/tests/__init__.py	 View		1 chunk	+5 lines, -2 lines	 0
> comments	
> > > Download
> > > A	 charmworld/tests/test_utils/__init__.py	 View		0 chunks	+-1 lines, --1
> > lines	
> > > 0 comments	 Download
> > > A	 charmworld/tests/test_utils/test_formatters.py	 View		1 chunk	+30
lines,
> -0
> > > lines	 0 comments	 Download
> > > A	 charmworld/tests/test_utils/test_ini.py	 View		1 chunk	+26 lines, -0
> lines	
> > 0
> > > comments	 Download
> > > M	 charmworld/tests/test_views/test_mainviews.py
> > > 
> > > 
> > > all of test_utils subpackage would just be in a module test_utils.py.
> > > 
> > > test_views would go to views/tests and s/test_mainviews.py/test_views.py
> > (what's
> > > a mainview ?) 
> > > 
> > > what do you think?
> > 
> > Definitely no problem relocating the tests into their modules. I personally
> > prefer more files with less code to make it easier to find the tests. I'd
> > suggest keeping the directories and just adding a tests module to views,
> utils,
> > etc if that's ok. 
> > 
> > Mainviews is definitely an admittingly poor name. I'll just dump those tests
> > back into the views/tests/__init__.py to match their location in
> > views/__init__.py. 
> > 
> > If it's ok, I'd like to land this with these changes and then have a follow
up
> > that reorg's so that it's just clear diff of moved files vs this with a lot
of
> > code changes.
> 
> sounds good re followup. re views/tests/__init__.py and views/__init__.py it
> would be nice if we could extract those files down to minimal contents and
place
> into appropriate modules, package __init__ files tend to be a be a hidden as a
> code location ie non obvious without inspection.

Ok, sounds like a plan. I'll work on getting the __init__'s emptied and into
filenames that sound better than mainviews. 

Thanks for the review.