[Prophet] Some random thoughts and questions on OpenResty for Prophet

[Jesse Vincent]

Hiya agentzh!

I'm sorry for the slooow reply.

On Sat  4.Oct'08 at 11:09:59 +0800, agentzh wrote:
> Hi, prophet folks!
> 
> I've been looking at Prophet and playing with SD for a while. I'm
> trying to get my head around it. I know it's still under active
> development and lacks documentation (severely). 

If you can list off things you want better docced, that can help steer
us toward improving those things.

> I just want to write
> down some things that I'd like to do with Prophet here in the hope of
> getting some guidance from the developers.
> 
> As one of the authors of OpenResty, I'd really like to hack up an
> OpenResty frontend for Prophet so as to make Prophet a database
> backend. It'll be great to be able to use our shiny admin site (
> http://openresty.org/admin/) to manage the local Prophet replicas as
> in ~/sd-bugs/. We can also look forward to more and more shiny web
> apps atop Prophet like a p2p BBS ;)


I'd really love that!

> OpenResty's Model API should be relatively easy. It seems the
> Prophet::Record class is the right way to go, but I'm still trying to
> grok all the API needed to achieve my goal. 

That seems about right. Prophet doesn't require that multiple objects of
the same type have the exact same schema, so we may have to be careful
adapting the Model API.

What methods do we need to implement to be able to support your model
API?

> For the View API, it seems
> that we need a SQL-like query language for Prophet (just like GQL for
> BigTable and FQL for Facebook). 

Well, we need a query language. "SQL-like" kind of makes me sad because
SQL is such a usability nightmare for developers, but I
wouldn't turn down a developer who wanted to work on something like that
:)


> Other APIs like Feed and Action API
> are trivial if we get models and views working.
>

*nod*

> Another interesting thing we've been thinking about is to use Prophet
> to serve as an (optional) auxiliary backend storage for the current
> PostgreSQL backend in OpenResty. 

Implementing alternate Prophet backends that store in a relational
database is definitely doable, but we'll have to be careful about
forcing exact schema on individual records.

> Data synchronization between  Pg and
> Prophet themselves deserves more thought.

In prophet, we have tools to support sync with a foreign database type,
but most often it makes sense to customize such things for the foreign
db's fixed schema.

> For me, I want Pg only to
> store the HEAD snapshot of the Prophet record base, so that we can
> enjoy cheap FTI (via tsearch2) and cheap table join and many other
> relational goodies. 

I'd really like those to be available to Prophet users without needing
to rely on a Postgres installation. One option here is to adopt some of
the work Yuval Kogman (nothingmuch) has been doing on KikouDB.

> For big data sets that Prophet can't handle very
> well, the Prophet support could be disabled via the OpenResty API by
> the user.
>

*nod*

> Well, yeah, I know it's also possible to make Pg a backend for
> Prophet. That's just another option, but (IMHO) makes very little
> sense to OpenResty itself though.

*nod* I think I'm most excited about making Prophet a sane backend for
OpenResty.  One option MIGHT be to let OpenResty use another "remote"
REST API as a backend store. That's something we've been considering
doing in Jifty forever.

Should we plan a hackathon on some of this stuff around the Beijing Perl
Workshop?


-jesse

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://lists.bestpractical.com/pipermail/prophet/attachments/20081017/7c8792c3/attachment.pgp